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TO:  Chief  of  Naval  Research,  Arlington,  Virginia  22217 

Attention:  Engineering  Psychology  Programs,  Code  455, 

Drs.  M.  A.  Tolcott  and  J.  J.  O'Hare 
FROM:  Dr.  John  A.  Swets 

Bolt  Beranek  and  Newman  Inc .  , 

Cambridge,  Massachusetts  02138 

SUBJECT:  Final  report  of  work  completed  under  the  support  of 

Contract  N000 1 4-76-C-0893 ,  Work  Unit  NR  196-145, 
between  Bolt  Beranek  and  Newman  Inc.  and  the 
Engineering  Psychology  Programs,  Office  of  Naval 
Research . 


I.  This  constitutes  a  final  report  of  work  completed  under  the 
support  of  Contract  N 000 1 4-76-C-0893 ,  Work  Unit  NR  196-145, 
between  Bolt  Beranek  and  Newman  Inc.  and  the  Engineering 
Psychology  Programs,  Office  of  Naval  Research.  The  contract 
was  initiated  1  June  1976  and  was  terminated  31  August  1979. 

II.  The  objective  of  this  contract  was  to  conduct  research  on 
perceptual  and  decision  processes  involved  in  the  detection, 
identification,  and  scaling  of  complex  visual  patterns,  with 
emphasis  on  temporal  aspects  of  the  patterns  and  processes. 
During  the  three  and  one-quarter  years  of  the  project,  five 
BBN  technical  reports  were  prepared  and  distributed,  each 
describing  a  substantial  experiment  or  set  of  experiments; 
all  five  have  been  or  will  be  published  in  the  archival 
literature.  A  book  chapter  not  distributed  as  a  BBN 
technical  report  was  also  prepared.  A  bibliography  of  these 
reports  is  appended.  Also  appended  is  a  list  of  individuals 
who  have  been  associated  with  the  project. 

III.  The  research  is  described  here  by  means  of  abstracts  of  the 
five  technical  reports  and  the  book  chapter.  A  concluding 
abstract,  prepared  for  an  international  congress,  summarizes 
the  main  part  of  the  work  done  under  the  contract. 
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1.  Signal  Detection  and  Ident i f icat ion  at  Successive 
Stages  of  Observation 

The  relationship  of  signal  identification  to  signal 
detection  was  examined  in  a  series  of  experiments.  The 
signals  were  idealized  lines  and  patterns  of  lines  in  a 
spectrographic  display.  On  each  trial,  progressively 
more  of  the  complete  spectrogram  was  exposed  in 
successive  observation  intervals  and  the  observer  made 
both  detection  and  identification  responses  after  each 
interval  that  were  based  on  the  accumulating  evidence. 
One  model  employed  conceives  of  detection  and 
identification  as  proceeding  together  over  time  as 
parts  of  a  unified  process.  A  second  model  employed 
shows  how  the  joint  detect  ion- and- ident if icat ion 
ROC  —  a  Relative  Operating  Characteristic  that  relates 
the"  joint  probability  of  correct  detection  and  correct 
identification  to  the  probability  of  a  false 
detection  --  may  be  predicted  from  the  simple  detection 
ROC.  Both  models  were  supported  by  the  data. 

(Technical  Report  3535;  published  in  Perception  & 
Psychophysics ,  1978,  2j[(U),  275-289.) 

2.  Identification  and  Scaling  of  Complex  Visual  Fatterns 

Experiments  were  conducted  to  examine  the  detection  and 
identification  of  complex  visual  patterns,  representing 
various  underwater  sounds,  in  terms  of  stimulus 
dimensions  indicated  by  a  psychological  scaling 
analysis  of  similarity  judgments.  Psychological 
dimensions  differed  somewhat  between  naive  and 
experienced  observers,  and  between  visual  and  auditory 
representations,  but  high  correlations  with  physical 
dimensions  were  found  in  all  cases.  A  second 
objective  of  the  study  was  to  relate  identification  to 
detection,  over  successive  stages  of  relatively  long 
signals.  The  ROC  analysis  developed  for  detection  of 
signals  in  random  noise  was  applied  to  discrimination 
of  wanted  and  unwanted  signals.  The  analysis  of 
stimulus  dimensions  enabled  qualitative  predictions  of 
the  relative  levels  of  detection  and  identification  for 
different  sets  of  wanted  and  unwanted  signals.  The 
"joint"  ROC,  based  on  the  probability  of  correct 
detection  and  identification,  provided  quantitative 
benchmarks  for  these  predictions. 

(Technical  Report  3536;  combined  with  Technical  Report 
3719  for  publication  as  "On  the  prediction  of 
confusion  matrices  from  similarity  judgments," 
Perception  &  Psychophysics ,  1979,  26(1),  1-19.) 


-3- 


On  The  Prediction  of  Conf usion  Matrices  from  Similarity 
Judgments 

This  study  of  human  identification  of  complex  visual 
stimuli  is  a  step  toward  an  integrated  description  of 
the  perceptual  process  whereby  stimuli  are  represented 
psychologically  and  the  decision  process  that  makes  use 
of  this  stimulus  representation  in  selecting  an 
identification  response.  A  multidimensional  scaling 
procedure  was  applied  to  judgments  of  stimulus 
similarity  to  derive  the  dimensions  of  a  perceptual 
space  and  the  relative  loci  of  stimuli  in  that  space, 
and  a  probabilistic  decision  model  based  on  weighted 
interstimulus  distances  was  developed  to  predict  the 
confusion  matrices  in  various  identification  tasks. 
The  high  accuracy  of  the  predictions  supplies  a  strong 
validation  of  the  use  of  multidimensional  scaling 
procedures  to  reveal  perceptual  structure,  in 
demonstrating  the  ability  of  that  structure  to  acccount 
for  behavior  in  an  independent  task.  From  the  other 
point  of  view,  the  empirical  success  of  this  approach 
suggests  a  relatively  simple  and  practical  means  of 
predicting,  and  possibly  enhancing,  identification 
performance  for  a  given  ensemble  of  visual  or  auditory 
stimuli . 

(Technical  Report  3719;  published  in  Perception  & 
Psychophysics ,  1979,  26^(1),  1-19.) 

The  Observer ' s  Use  of  Perceptual  Dimensions  in  Signal 
Identification 

We  examine  a  model  of  the  process  of  stimulus 
identification,  which  assumes  that  complex  visual  or 
auditory  stimuli  are  represented  as  vectors  in  a 
multidimensional  perceptual  space,  and  which  postulates 
a  simple  probabilistic  decision  process  based  on  the 
geometric  structure  of  the  perceptual  space.  We 
present  evidence  from  several  conditions  of  an 
identification  task  that  human  observers  engage  in  a 
continuing,  dynamic  process  in  which  dimension  salience 
weights  are  tuned  to  optimize  identification 
performance.  In  addition,  we  verify  the  reliability  of 
the  INDSCAL  multidimensional  scaling  procedure  in 
deriving  the  geometric  structure  of  the  observers' 
perceptual  space  for  the  set  of  visual  spectrograms 
used  in  our  identification  tasks.  We  also  present 
evidence  supporting  an  assumption  of  dimensional 
decomposability  made  in  the  decision  process.  Finally, 
we  observe  that  the  model  is  successful  in  accounting 
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for  approximately  90  percent  of  the  variance  in 
individual  confusion  matrices,  averaged  over  18 
observers  x  conditions. 

(Technical  Report  3930;  to  appear  in  Attention  and 
Performance  VI II ,  R.  S.  Nickerson  (ed.l  Hillsdale, 
FT  771  Lawrence  Erlbaum  Associates,  in  press.) 

5.  Identification  of  Confusible  Curv ilinear  Shapes : 
Prediction  of  Indiv idual  Confusion  Matrices 

Identification  confusion  matrices  were  obtained  from 
three  observers  for  a  set  of  nine  closed,  curvilinear 
shapes.  These  stimuli,  biological  in  appearance, 
varied  along  two  nonverbalizable  physical  dimensions 
related  to  the  angles  of  rotation  of  two  of  six 
underlying  component  forms.  Two  perceptual  dimensions, 
each  linearly  related  to  one  of  the  two  physical 
dimensions,  were  derived  from  multidimensional  scaling 
of  pair-wise  similarity  judgments.  Using  this  derived 
perceptual  space  and  a  probabilistic  decision  model,  we 
were  able  to  account  for  95  percent  of  the  variance  in 
individual  observer's  confusion  matrices. 

(To  appear  as  Technical  Report  4196;  to  be  submitted  to 
Percept  ion  &  Psychophysics  . ) 

6.  Perceptual  Spaces  Revealed  by  Multidimensional  Scaling 
Procedur  es 


The  development  of  multidimensional  scaling  (MDS) 
procedures  in  recent  years  has  allowed  us  to  pursue  the 
concept  that  a  complex  auditory  or  visual  stimulus  can 
be  represented  psychologically  as  a  point  in  a 
multidimensional  perceptual  space.  In  this  paper  we 
review  the  literature  which  has  applied  MDS  procedures 
to  similarity  judgments  and  to  identification  confusion 
matrices  to  obtain  information  both  about  the  set  of 
dimensions  underlying  perception  of  a  set  of  complex 
stimuli  in  the  multidimensional  space.  We  also  raise 
the  question,  what  should  we  require  of  a 
multidimensional  representation  to  consider  it  valid? 
We  suggest  one  criterion  —  prediction  of  independent 
data  —  and  present  a  model  and  two  sets  of  data  which 
illustrate  the  point. 

(To  appear  in  "Auditory  and  Visual  Pattern  Recognition , 
D.  J.  Getty  and  J .  H.  Howard,  Jr"  ( eds .  )  Hillsdale , 
N.  J.:  Lawrence  Erlbaum  Associates,  in  press.) 
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Perceptual  Dimensions :  Similarity  3nd  Identification 

We  describe  a  view  of  stimulus  identification'  that 
assumes  (1)  that  complex  visual  or  auditory  stimuli  are 
represented  perceptually  as  vectors  in  a 
multidimensional  Euclidean  space,  and  (2)  that  the 
probabilities  of  identification  responses  are 
determined  by  relative  inter st imul us  distances  in  the 
perceptual  space.  In  practice,  we  derive  the 
dimensions  of  the  perceptual  space,  and  the  loci  of  the 
stimuli  within  the  space,  from  application  of  a 
multidimensional  scaling  procedure  to  pairwise 
judgments  of  stimulus  similarity.  Based  on  the  derived 
interstimulus  distances,  we  then  predict  the 
identification  confusion  matrix  for  individual 
observers  by  means  of  a  probabiliistic  decision  model. 
In  a  series  of  three  experiments,  the  model  has 
accounted  for  an  average  of  more  than  90  percent  of  the 
observed  capability  in  33  individual  confusion 
matrices . 

Three  different  sets  of  complex  visual  stimuli  have 
been  used:  (1)  a  set  of  27  synthesized  two-formant 
spectrograms,  suggestive  of  speech  spectrograms,  which 
varied  along  three  physical  dimensions,  (2)  a  set  of  8 
spectrograms  of  real,  underwater  sounds,  and  (3)  a  set 
of  9  shapes,  organic  in  appearance,  that  varied  along 
two  nonverbalizable  physical  dimensions.  In  each  case, 
we  derived  a  perceptual  space  that  could  be  directly 
related  either  to  the  manipulated  physical  dimensions 
(sets  1  and  3)  or  to  physical  measures  derived  from  an 
analysis  of  the  signals  (set  2). 

Our  decision  model  assumes  that  perceptual  dimensions 
are  not  necessarily  weighted  equally  in  determining 
interstimulus  distance,  and  that  the  relative 
magnitudes  of  these  dimension  salience  weights  may 
differ  across  observers.  Furthermore,  we  suggest  that 
these  weights  may  be  adaptively  tuned  by  an  observer  in 
order  to  optimize  identification  performance,  the 
criterion  depending  upon  both  the  task  and  the  stimulus 
set.  In  an  experiment  in  which  observers  identified  a 
subset  of  four  "signals,"  among  a  set  oi  eight  stimuli, 
with  a  different  subset  of  four  defined  as  signals  in 
each  of  3  conditions,  we  observed  that  the  patterns  of 
dimension  salience  weights  varied  across  conditions. 
Moreover,  each  observed  pattern  of  weights  was 
generally  in  agreement  with  the  theoretical  pattern  of 
weights  that  would  maximize  the  probability  of  correct 
identification.  We  have  also  demonstrated  the  dynamic 
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properties  of  the  tuning  process  in  another  experiment 
in  which  the  relative  usefulness  of  a  particular 
dimension  was  drastically  reduced  from  the  first 
condition  of  the  experiment  to  the  second.  Analyzing 
successive  blocks  of  trials  within  the  second 
condition,  we  observed  the  pattern  of  dimension  weights 
to  change  gradually  for  several  observers,  from  near 
optimum  for  the  first  condition  to  near  optimum  for  the 
second  . 

The  use  of  multidimensional  scaling  procedures  to 
reveal  perceptual  structure  receives  substantial 
validation  from  our  results,  in  demonstrating  the 
ability  of  that  structure  to  account  for  ident i f icat ion 
performance  in  an  independent  task.  At  the  same  time, 
we  gain  confidence  in  the  decision  model,  in 
demonstrating  its  predictive  accuracy  based  on  an 
independently  derived  perceptual  space.  Finally,  this 
approach  suggests  a  relatively  simple  and  practical 
means  of  predicting,  and  possibly  enhancing, 
identification  performance  for  a  given  set  of  visual  or 
auditory  stimuli. 

(This  abstract  will  appear  in  the  Proceedings  of  the 
XXIInd  International  Congress  of  Psychology,  Leipzig, 
1980.  ) 


r 


Personnel 


John  A.  Swets 
Principal  Investigator 

David  J.  Getty 
Senior  Scientist 

David  M.  Green 
Senior  Scientist 

Joel  B.  Swets 
Research  Assistant 

Barbara  N.  Freeman 
Programmer 

Terry  Timberlake-Kinter 
Research  Assistant 


Mildred  Chapman  Webster 
Project  Secretary 


-8- 


Bibliography 

Swets,  J.  A.,  Green,  D.  V.,  Getty,  D.  J.,  and  Swets,  J.  B. 
Signal  detection  and  identification  at  successive  stages  of 
observation.  BBN  Technical  Report  3535,  July,  1977.  Also 
published  in  Perception  4  Psychophysics ,  1978,  23>9),  275-289. 

Swets,  J.  A.,  Green,  D.  M.,  Getty,  D.  J.,  and  Swets,  J.  B. 
Identification  and  scaling  of  complex  visual  patterns.  BBN 
Technical  Report  3536,  November,  1977.  Combined  with  Technical 
Report  3719  for  publication  as  "Cn  the  prediction  of  confusion 
matrices  from  similarity  judgments,"  in  Perception  & 
Psychophysics ,  1979,  26(1),  1-19. 

Getty,  D.  J.,  Swets,  J.  A.,  Swets,  J.  B. ,  and  Green,  D.  M.  Cn 
the  prediction  of  confusion  matrices  from  similarity  judgments. 
BBN  Technical  Report  3719,  October  1977.  Also  published  in 
Perception  4  Psychophysics ,  1979,  26_(  1  )  ,  1-19. 

Getty,  D.  J.,  Swets,  J.  B.,  and  Swets,  J.  A.  The  observer's  use 
of  perceptual  dimensions  in  signal  identification.  BBN 
Technical  Report  3930,  June,  1979.  To  appear  in  Attention  4 
Per  form  ance  VI 1 1 ,  R.  S.  Nickerson  ( ed . )  Hillsdale,  N.  J.: 
Lawrence  Erlbaum  Associates,  in  press. 

Getty,  D.  J.  Perceptual  spaces  revealed  by  multidimensional 
scaling  procedures.  To  appear  in  Auditory  3nd  Visual  Pattern 
Recognition  ,  D.  J.  Getty  and  J.  H.  Howard,  Jr.,  ( eds . ) . 
Hillsdale,  N.  J.:  Lawrence  Erlbaum  Associates,  in  press. 

Getty,  D.  J.,  and  Swets,  J.  A.  Perceptual  dimensions:  Similarity 
and  identification.  To  appear  in  the  Proceed ings  of  the  XXI Ind 
International  Congress  of  Psychology,  Leipzig  ,  i960 . 


